Photography is the process of making pictures by means of the action of light. Light is the commonly used term for electromagnetic radiation in a frequency range that is visible to the human eye. Light patterns reflected or emitted from objects are recorded by an image sensor through a timed exposure. Image sensors can be chemical in nature, such as photographic film, or solid state in nature, such as the CCD and CMOS image sensors employed by digital still and video cameras.
Digital cameras have a series of lenses that focus light to create an image of a scene. But instead of focusing this light onto a piece of film, as in traditional cameras, it focuses it onto the image sensor which converts the electromagnetic radiation of the light into an electrical charge. The image sensor is said to be a picture element, or a ‘pixel.’ The electrical charge indicates a relative intensity of the electromagnetic radiation as perceived by the image sensor, and generally is used to associate a light intensity value with the pixel.
FIG. 1 illustrates typical component blocks that may be included in a digital image processing system 10. The system 10 includes a signal source 100 and a signal processing chain that consists of integrator 110, analog to digital converter (ADC) 120 and DSP 130. Signal source 100 could for example be a sensor such as a light intensity sensor that generates an electrical response in response to electromagnetic radiation, such as light, impinging upon it.
The output of integrator 110, VOUT, is input to ADC 120. ADC 120 performs the analog to digital conversion function. The analog to digital conversion function is well known in the art. The analog signal VOUT present at ADC 120 input is converted into signal VD that can take one of a set of discrete levels.
The quality of the signal is improved by integrator 110 which integrates the signal VIN. FIG. 2 illustrates the nature of the signal improvement. Waveform 200 is the combination of a constant value signal generated by signal source 100 and additive noise that corrupts the constant value signal. Waveform 210 is the integrator output generated in response to input signal waveform 200. It is readily observed that signal fluctuations caused by the additive noise decrease in waveform 210.
Signal source 100 could be a light intensity sensor that is used in a timed application, such as in a digital camera application where the sensor is exposed to the light for a specific duration of time, commonly referred to as the exposure time. The integrator 110 then also serves the function of integrating the response of sensor 100 caused by all photons received during the exposure time into one value, such as for example a voltage, to be read-out at the end of the exposure time.
FIGS. 3 illustrates a typical image sensor circuit. Signal source 1000 is a light sensor that by way of example can be said to be a photodiode. Capacitor 1040 is a simple integrator. The input to the integrator is the output of signal source 1000. Capacitor 1040 is reset by switch 1050 which is in the closed position prior to starting the integration process. At the start of the integration process switch 1050 opens and the voltage across capacitor 1040 begins to change in response to the input signal originating from signal source 1000. At the end of the integration process switch 1030 closes and integrator output 1060, VOUT, is sampled. FIG. 3 is an illustrative diagram. The implementation of other similar integrators with identical functionality is well known to one skilled in the art.
Integrator output 1060, VOUT, cannot in general exceed the upper limit imposed by the available power supply voltage. Power supply voltages are decreasing in state-of-the-art equipment due to stringent power consumption requirements. Integrator output 1060 cannot exceed the power supply voltage and will saturate if the integrator output signal continues to build after reaching the power supply voltage level. The saturation condition is illustrated in FIG. 4A. Saturation occurs when the output voltage reaches the available power supply voltage and is unable to increase any further in response to the input signal. Signal saturation causes system performance degradation. FIGS. 4A through 4C illustrate potential distortions at the output of a pixel structure consisting of light sensor 100 and integrator 110 due to the dynamic range limitation of the photosensitive element structure and more specifically of the integrator structure.
Segment (a) of FIG. 4A illustrates the linear increase of integrator 110 output in response to a constant input signal of different level. The image sensor structure will perform well for the range of input light intensities that give rise to the linear output of segment (a); the image sensor structure will not perform well for the range of input light intensities that give rise to the saturated output of segment (b).
The integrator output response is indicative of limited dynamic range. As illustrated in FIG. 4A the image sensor will render well shadow detail but will fail to render highlight detail. It is possible to shift the response as illustrated FIGS. 4B and 4C. In FIGS. 4B and 4C the dynamic range of the image sensor remains the same but the response characteristic is shifted. The response characteristic of FIG. 4B loses shadow and highlight detail but retains good midrange response. The response characteristic of FIG. 4C loses shadow detail and partial midrange detail in order to maintain good highlight detail.
FIG. 5A illustrates the histogram of the pixel intensities of an overexposed image capture where a multitude of pixels were driven into saturation, such as in FIG. 4A. As seen in FIG. 5A the maximum pixel structure output value is ‘255’ and the units used are the ADC 120 output codes corresponding to the pixel output voltage. The light intensity caused many light sensors 100 to output a value that saturated the integrator 110 as the exposure progressed during the exposure period. The maximum (saturated) value of the integrator 110 output caused the ADC to generate the output code ‘255’ which is the maximum output code for an 8-bit ADC. The image capture will be of suboptimal quality due to the inability of those pixels subject to high intensity light inputs to achieve a sufficiently high output level. A lower integrator 110 gain would have caused the outputs of the light image sensor subject to high intensity light inputs to register a below −255 output and avoid the high end distortion.
FIG. 5B illustrates the histogram of the pixel intensities of an underexposed image capture where a multitude of pixels were not exposed to sufficient light to achieve a minimum output value. As seen in FIG. 5B the minimum pixel structure output value is ‘0’ and the units used are the ADC 120 output codes corresponding to the pixel output voltage. The light intensity caused many light sensors 100 to output a value that failed to cause integrator 110 to output a sufficiently high value to cause a minimal ADC output code as the exposure progressed during the exposure period.
The image capture will be of suboptimal quality due to the inability of those pixels subject to low intensity light inputs to achieve a sufficiently high output level. The distortion illustrated in the histogram of FIG. 5B corresponds to the individual pixel distortion of FIG. 4C. A higher integrator 110 gain would have caused the outputs of the light image sensor, subject to low intensity light inputs, to register an above-zero output and avoid the low end distortion.
FIG. 6 illustrates the response curve of a pixel structure built using double-slope technology. The nonlinear extension of dynamic range illustrated in FIG. 6 avoids saturation effects; however, the non-linear relationship between the intensity of the electromagnetic energy impinging upon the sensor and the sensor's output causes the image to be captured with reduced resolution when high levels of light intensity are present.
Other approaches such as multiple exposure combining, conditional slope switching and logarithmic response pixel structures have been published. The multiple exposure combining, conditional slope switching and logarithmic response pixel structures exhibit performance degradations that render them unsuitable for high performance image acquisition tasks.
Integrator saturation is the limiting factor in the dynamic range performance of a pixel structure. Solutions to the integrator saturation problem have been published. One feature the published solutions have in common is the monitoring of the integrator output to detect the onset of saturation condition at which time the integrator is discharged and the event is recorded. This class of solutions is difficult to implement efficiently in integrated circuits (ICs) due to accuracy requirements of analog components and non-standard analog implementations. The implementation of accurate comparators that operate in a noisy environment near the power supply voltage, where integrator outputs begin to saturate, is a difficult undertaking that consumes excessive power, an undesirable operational feature.
Analog IC designs are difficult and time consuming to implement. It is advantageous to use standard building blocks that have been fully debugged and optimized for size, power consumption and performance. The class of published solutions does not meet this requirement.